NILPOTENT GELFAND PAIRS 
AND SPHERICAL TRANSFORMS OF SCHWARTZ FUNCTIONS 
I. RANK-ONE ACTIONS ON THE CENTRE 

VERONIQUE FISCHER, FULVIO RICCI, OKSANA YAKIMOVA 

O 

Abstract. The spectrum of a Gelfand pair of the form (K x AT, K), where N is a nilpotent group, can be 
embedded in a Euclidean space M. d . The identification of the spherical transforms of A"-invariant Schwartz 
fl^ functions on N with the restrictions to the spectrum of Schwartz functions on R d has been proved already 

when N is a Heisenberg group and in the case where N = Ns t 2 is the free two-step nilpotent Lie group with 
three generators, with K = SO3 [2l [3l [TT]. 

We prove that the same identification holds for all pairs in which the X-orbits in the centre of N are 
spheres. In the appendix, we produce bases of J\-invariant polynomials on the Lie algebra n of N for all 
Gelfand pairs (K x N, K) in Vinberg's list l27ll30l . 
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1. Introduction 



Let N be a connected, simply connected, nilpotent Lie group and A a compact group of automorphisms 
of N. We are interested in the situation where (A K N,K) is a Gelfand pair, i.e., where the convolution 
algebra L 1 (A^) K of A-invariant integrable functions on N is commutative. 

To simplify the notation, under these circumstances we will say that (N, A) is a nilpotent Gelfand pair, 
n.G.p. in short. This notion is equivalent to that of "commutative nilmanifold" of [281 Chapter 13]. 

According to Vinberg's reduction theorem [2B], the general Gelfand pair (G,K), with G, A Lie groups, 
is built up over two elementary constituents, a reductive G.p. and a nilpotent G.p. The classification of 
nilpotent Gelfand pairs was obtained in [57J 130] , see also [2H1 Chapters 13,15]. We point out that in every 
n.G.p. N is at most step-two [4]. 

The Gelfand spectrum E(iV, A) of L}{N) K admits natural homeomorphic embeddings in Euclidean spaces 
|10j . each associated with a finite d-tuple T> = [D\, . . . , Dd) of self-adjoint generators of the algebra V)(N) K 
of left- and A-invariant differential operators on N. The embedded image £x> C K d of E(JV, A) is obtained 
by assigning to each bounded spherical function ip the d-tuple (fix(<p), . . . , Hd(<p)) of eigenvalues of <p as an 
eigenfunction of Di, . . . , Dd- 

It has been proved in several cases that the Gelfand transform establishes the isomorphism 

(1) S{N) K - S(Jh) = S(R d )/{f : = 0} , 

and that this condition is independent of the generating set T> as above, cf. [3J, Section 3. More precisely, 
consists of two separate statements: 

Given u G <S(R d ), there is a unique F G S(N) K such that u — QF on Ex>, 
and F depends continuously on u. 

Given F G S(N) K and peN, there are q = q(p) G N and u p G S(R d ) 
such that QF — u p on E-jj and ||w p ||(p) < C p ||-F||( g ). 

Since |2]) holds on any n.G.p., cf. (3] [11] , the main problem is to prove This has been proved in [21 [3] 
for any pair (N,K) in which N is the (2n + 1) -dimensional Heisenberg group H n , and in [TT] for the case 
where N is the free two-step nilpotent group with three generators and K = SO3 or 3 . 

Condition ([T]) has an additional interpretation from the point of view of functional calculus on the operators 
Dj. The set Sx> is the joint L 2 -spectrum of D\. . . . , Dd, and, given a bounded Borel measurable function m 
on Ex>, the inverse Gelfand transform of m is the convolution kernel K m of the operator m{D\, . . . ,Dd) G 

Then ([!]) is equivalent to saying that K m G S(N) K if and only if m G <S(Ex>). This connection with 
functional calculus is more than just a remark, because it gives a quick proof of ([2]), which in turn intervenes 
in the proof of 

The problem of characterising the Gelfand transforms of A"-invariant Schwartz functions on the Heisenberg 
group has been addressed, and solved, by other authors in the past, cf. [13) and, in greater generality, [5]. 
We find that the conditions given, though explicit, are quite technical, as they are expressed in terms of the 
different parameters identifying bounded spherical functions, and do not suggest a unifying idea for more 
general cases. 

In this paper we present a further step in a general attempt to generalise (JlJ to other nilpotent Gelfand 
pairs. 

Let 3 be the centre of the Lie algebra n of N. Then the action of K on n restricts to 3. We say that 3 is 
rank-one if a generic A-orbit in 3 is of codimension one. 

Since the action of A on n also restricts to the derived algebra [n, n] C 3, our assumption implies that 
[n, n] = 3, i.e., n does not contain abelian factors. Let denote a A-invariant complement of [n, n] in n. Then 
[0,0] =3. We will also assume that A acts irreducibly on 0. 

Our main result is the following. 
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Theorem 1.1. Assume that 3 is rank-one and is irreducible. Then ([2]), ([3|, and hence ([!]) hold. 

We disregard the cases where N is abelian, already discussed in [3J, and, for the same reason, the cases 
where 3 is one-dimensional (i.e., N is the Heisenberg group). 

The list of all n.G.p. with 3 rank-one and D irreducible can be derived directly from Vinberg's list, cf. 
[27} 130]. In order to present it in a simplified form, we first state a preliminary "if-reduction theorem", 
which extends Section 8 of [3J and which will be proved in Section [3] 

Theorem 1.2. Assume that {N,K\) and (N,K%) are both nilpotent G elf and pairs, and that Kx<K%. If 
holds for (N, K\), it also holds for (JV, K-z). 

We can then limit our attention to the n.G.p. with 3 rank-one, irreducible and K minimal, in the sense 
that (N,Ko) is not a n.G.p. for any normal subgroup K of K. The list of such pairs is in Table [l] 

Like in the previous proofs of ([I]) , a crucial step is the proof of a "Geller-type formula" , which provides 
the a-priori Taylor expansion for the desired Schwartz extension of QF on the singular part of E-p (as defined 



in Subsection 4.2) 



With respect to the Heisenberg group case, a serious technical complication occurs for the pairs listed in 
the second and third block of Table [I] (in fact a simpler proof, based on a generalisation of Theorem 1.2 



is 



possible for the pairs in the first block). This complication is due to the presence, in any set of generators of 
the algebra of /^-invariant polynomials on n, of "mixed polynomials" , i.e. polynomials depending on both 
the 0- and the 3-variables. By symmetrisation, this has the effect that "mixed differential operators" are 
present in any choice of T>. 

Mixed invariants were already present in the case treated in (TT|, i.e., line 10 of Table [I] The method used 
there, based on an "exact Geller-type formula" (Proposition 5.2 and Corollary 5.3 in [TTfland a radialisation 
argument, turns out to be not so easily extendable to the other pairs (even though the validity of such a 
formula is implied a posteriori by Theorem 



We prove instead a weaker "Geller-type formula with remainder term" (Proposition 5.1) which is easier 
to prove and good enough for our purposes. The radialisation argument is replaced by the extension process, 
modeled on [3] and presented in section [6] 

As we have mentioned already, our approach requires minimal information on the spherical functions 
involved in each case and on the parameters they depend on. Reductions to quotient groups that are 
Heisenberg or TJ-type and Radon transform arguments are sufficient to provide enough information when 
needed. 

However, we require full knowledge of which pairs are involved and how Hilbert bases (i.e., systems of 
generators for the algebra of if-invariant polynomials on n) look like, cf. Table [2j 

The proofs required to obtain the Hilbert bases in Table [2] are given in the Appendix. For reasons of 
completeness, and in view of future extensions, Hilbert bases are given not only for the pairs considered in 
this paper, but for all pairs in Vinberg's list (i.e. with irreducible, but no restrictions on [n, n]). Surprisingly, 
in all cases the algebra ^[n]^ of polynomial Zf-invariants on n admits an algebraically independent system 
of generators. 



2. Preliminary facts 



In this section we give a quick introduction to nilpotent Gelfand pairs and to the tools used in the sequel. 
We refer to [28] for uncommented statements concerning the general theory of Gelfand pairs. 

2.1. Spherical functions and Gelfand transform. 

If K is a compact subgroup of Aut(N), the space L}{N) K of fC-invariant integrable functions on N is an 
algebra under convolution. 
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Assume here that L 1 (A r ) ii " is commutative. This implies that the nilpotent Lie group N is of step at most 
two [3]. The K-spherical functions on TV are the smooth functions tp on N such that 

(i) ^(0) = 1; 

(ii) <p is an eigenfunction of all operators in B>(N) K . 

For <p spherical and D G ©(JV)^ , we call hd{v) the eigenvalue of (/? when acted upon by D. 

The multiplicative linear functionals on L 1 (N) K are given by integration against the bounded spherical 
functions. So the Gelfand spectrum E(JV, if) of L 1 (iV)- K ' can be identified with the set of bounded spherical 
functions. Under this identification, the Gelfand topology on E(JV, K) coincides with the compact-open 
topology on the set of bounded spherical functions. 

The Gelfand transform of a function F G L 1 (N) K is the function QF G Cq(T,(N, K)) given by 

GF{tp) = [ F(x)(p(x~ 1 ) dx . 
Jn 

As a consequence of the symmetry of L X (N) [IS], the bounded spherical functions are of positive type. 
An integral formula expressing them in terms of the irreducible unitary representations of N is given in [4] . 

Given an irreducible unitary representation ir of N and k £ K, let ir k be the representation given by 
TT k (n) = 7r(fc _1 n) for n G iV. Denote by K n the subgroup of those elements k £ K such that 7r fc ~ n. 

The unitary intertwinings operators give a projective unitary representation ov of -K^- on the representation 
space of 7r. Then H T decomposes into irreducible if^-invariant subspaces without multiplicities (this 
condition, for generic tt, is in fact equivalent to saying that (N,K) is a n.G.p.). Let 

= Kr,cr 

be this decomposition, with Vn,o possibly trivial. 

Theorem 2.1. Assume that V v a is nontrivial. Given a unit vector v G a , set 

(4) f7r.a( x ) = / (Tr(kx)v, v) dk . 

Jk 

Then 

(i) ^7r,(T is K-spherical, bounded, and does not depend on v; 

(ii) i/ 7r' ~ 7r fc /or some fc G K , then ip-x,a — fn'.a f or every a G K K) modulo the natural identification of 

with K^i ; 

(iii) given any bounded K-spherical function tp, there exist 7r G N, unique modulo the action of K, and a 
unique a G K^ with V n ^ nontrivial, such that ip = ip Wta . 

See [551 Section 14.5] for a description of this result in terms of spherical representations of the semi-direct 
product K k N and [7] for their correspondence with coadjoint orbits of K X N. 

2.2. Embeddings of S(JV, K) in Euclidean spaces. 

Let n be the Lie algebra of N, 6(n) its symmetric algebra, il(ti) the universal enveloping algebra of n. 
The identification of il(n) with the algebra D(iV) of left-invariant differential operators on N will be assumed 
throughout. 

We will use a modified symmetrisation operator A' : ©(n) — > il(rt), related to the standard symmetrisation 
A by the formula 

(5) A'OO = r^\(x a ) , 

for every monomial x a G ©(n). This modification has the effect that the differential operator D = X'(P) is 
formally self-adjoint, i.e. 

/ Df(x)gjx)dx= [ f(x)DgJx)dx, V /, g G (N) , 
Jn Jn 
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if and only if P is real- valued. 

Since A' is i^-equi variant, it establishes a linear isomorphism between the if-invariant subalgebras &(n) K 
and D(N) K . Even though both subalgebras are commutative, A' is not an algebra isomorphism. Nevertheless, 
if V C &(n) K is a generating set, the same is true for A'('P) in V)(N) K . Then Hilbert's basis theorem implies 
the existence of finite generating sets of V>(N) K . 

In the sequel we will need to consider finite ordered sequences (Di, . . . , D d ) of elements of B(N) K such 
that the Dj form a generating set. For this reason, we call Hilbert basis a finite sequence p — (pi, . . . , pd) of 
real- valued if-invariant polynomials which generate &{n) K . Clearly p defines a map 

p : n — > R d , 

called the Hilbert map. 

To any Hilbert basis p — (p%, . . . , pd) we associate the d-tuple T> = (l?, . . . , Dd) of generators of V)(N) K , 
where Dj = X'(pj). 

We denote by D a K- invariant complement of [n, n] in n. Hilbert's basis Theorem also implies the existence 
of a bi-homogeneous Hilbert basis of n, i.e., homogeneous in each of the variables v € and z £ [n, n]. Note 
that if a polynomial P on n is bi-homogeneous, then its symmetrisation A' (P) is a homogeneous differential 
operator where the homogeneity of operators on N is referred to the automorphic dilations (v, z) i-> (rv, r 2 z) 
of N. Furthermore, if P is of degree 7' and 7" in v and z respectively, then A'(P) is of degree 7 = 7' + 27". 

Thus we always have the possibility of constructing a Hilbert basis organised as follows: we start with 
a Hilbert basis p„ of if -invariant polynomials on containing the square norm \v\ 2 on 0, and a Hilbert 
basis p 3 of if-invariant polynomials on [n, n]; then we complete it, if necessary, with an additional set /9 8 ,[n,n] 
of polynomials depending on variables from both D and [n, n]. By symmetristion, this leads to a family of 
homogeneous generators oiV)(N) K containing the hypoelliptic operator A'd^l 2 ). 

Let P £ ©(n) be a real-valued polynomial, and D = A'(P). If tt E N, dir(D) operates on each K^- 
invariant subspace V^^ as poi^Tr^)!- Since V^^ is finite-dimensional and dir(D) is symmetric, this implies 
that poi'-P-K.a) is real. 

The following theorem is contained in |10j . 

Theorem 2.2. Let T> = (D\, . . . , Dd) a generating d-tuple of formally self-adjoint elements ofH)(N) K , and 
let 

(6) £r> = { (p Dl (<p),..., p Dd (<p)) : V € E(JV, K)} c R d . 
Then Ep is closed in R d and the map 

(7) $p:<£>i — > (p Dl (<f),. . . ,p Dd (f)) 
is 1-to-l and a homeomorphism ofH(N,K) onto £x>. 



2.3. Schwartz spaces. 

For functions on ]R n , the p-th Schwartz norm is 
(8) \\f\k P )= E sup (1 + 1^1^/(^)1 , 

and 5(R") is the space of C°°-functions / for which ||/||( p ) < 00 for all p £ N, endowed with the Frechet 
topology induced by the family of all Schwartz norms. Since this topology does not depend on the choice of 
coordinates, the Schwartz space can be defined intrinsically on any finite-dimensional real vector space. 
Using the fact that the exponential map exp : n — > N is a diffeomorphism, we define 

S(iV) = {/:/oexpeS(n)} ■ 
We refer to [12] for the basic facts concerning S(N), in particular about the possibility of defining S(N) 
intrinsically, with partial derivatives replaced by left-invariant vector fields and Euclidean norm replaced by 
a homogeneous norm in (fsb . 
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If E is a closed subset of Mr, we call S(E) the space of restrictions to E of functions in 5(IR <i ), with the 
quotient topology induced from the identification 

S{E) = S(R d )/{f :/ = 0oni?}. 

The following statement is a consequence of Lemma 3.1 in [3J. 

Proposition 2.3. Let T> = (Di, . . . , D c i) and T>' = {D[, . . . , D' d ,) two generating families of elements of 
D(N) K . Then the map 

/— >/o(3i>o^, 1 ) 
is an isomorphism from <S(Ex>) onto <S(E-p/). 

This implies that the validity of ([I]) for a given n.G.p. does not depend of the choice of V. 

2.4. Spectral multipliers. 

Let T> = {Di, . . . , Dd) be a generating system of formally self-adjoint elements of D(JV) . Then the Dj 
are essentially self-adjoint and Ex> is their joint L 2 -spectrum (cf. [II] . Proposition 3.1). 

Therefore, any bounded Borel function m on E-p defines a bounded operator m(Di, . . . , Dd) on L 2 (N) 
which is left-invariant and commutes with the action of K. By the Schwartz kernel theorem, there is 

K m =m{D 1 ,...,D d )5 £S'{N) K , 

such that 

m{D 1 ,...,D d )F = F*K m . 
The following proposition combines together Theorem 5.2 in [3J and Proposition 3.3 in [IT] . 

Proposition 2.4. Let D\, . . . ,D^. dT> be homogeneous, with at least one Dj hypoelliptic. 

If g G 6>(]R fc ) ; then g(Di, . . . , Dk)So € S(N) K . Furthermore the mapping g i — > g(D\, , . . ,D).)8q is 
continuous from S(R k ) to S(N) K . 

Notice that this statement proves |2]) for V consisting of homogeneous operators, with one of them 
hypoelliptic. We can always choose such T> (see subsection 2.2). Thus by Proposition 2.3 this gives © for 
every n.G.p. 



2.5. Abelian pairs. 

We briefly discuss the case of a n.G.p. where TV is abelian. Without loss of generality, we can assume 
that N = Mr and that if is a closed subgroup of O n . Proofs and details on the following statements can be 
found in [3J. 

If p = {px, . . . , pd) is a Hilbert basis, the d-tuple T> consists of the constant coefficient operators Dj = 
Pj(« _1 V), where V = {d Xl , . . . , d Xn ). Then 

E p = p{M n ) . 

The Gelfand transform of a function F G L 1 (W 1 ) K and its Fourier transform F are related by the identity 

(9) F = GFop. 

For abelian pairs, Theorem |l.l| is a consequence of ^ and of the following result, adapted from [24l 119] , 

Theorem 2.5. There exists a continuous linear operator Ek ■ S{M n ) K — > S{M. ) such that, for every 
G g S{W l ) K , 

G = {£ K G) o p . 

Corollary 2.6. The map F i — > £kF is linear and continuous, and its composition with the canonical 
projection of S(M. d ) onto iS(E-p) is Q. 

This shows that ^ holds with u p = SrF. This is a strong improvement on because u p is independent 
of p and it depends linearly on F. 
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2.6. H-type groups and 0-radial functions. 

An H-type Lie algebra is a step-two nilpotent Lie algebra n = D ©3, endowed with an inner product ( , ) 
such that 

(1) 3 = [n,n]; 

(2) t>±3; 

(3) for any »et) and the element J z v defined by the condition 

(10) (J z v,v'} = (z,[v,v']) , Vw'eo, 
satisfies the equality 

(11) \J z v\ = \z\\v\. 

An iJ-type group is a connected, simply connected Lie group whose Lie algebra is H-type. 

It is proved in [5] that the space Ll(N) of 0-radial integrable functions F on any iJ-type group N (i.e. 
such that F(v, z) = F(v', z) if \v\ — \v'\) is a commutative Banach algebra under convolution. 

The following statement combines results from jU] and to show that L\{N) admits a spectral analysis 
that resembles very much the spherical analysis of nilpotent Gelfand pairs. One must take into account that, 
in general, L\(N) does not coincide with L 1 {N) K for any K. 

Theorem 2.7. Let {X%, . . . , X m }, {Ti,...,T„} be orthonormal bases of D and 3 respectively (regarded as 
left-invariant vector fields on N). Set L = — Then 

(1) the multiplicative linear functionals on L\(N) are given by integration against the bounded "spherical" 
functions, defined as the bounded V-radial joint eigen] 'unctions ip of T> = {L, i T%, . . . , i _1 T„} such 
that ip(0) = 1; 

(2) the Gelfand spectrum of L\(N) is homeomorphic to the set S-p of (n + l)-tuples of eigenvalues of 
the bounded spherical functions, relative to the elements of T>; 

(3) is also the joint L 2 -spectrum of the elements ofT>; 

(4) the analogue of ([T]) holds, i.e., under the Gelfand transform, 

3. Proof of Theorem 11.21 



Let K\ be a subgroup of K2, and assume that (N,Ki) is a nilpotent Gelfand pair. Then also (N, K2) is 
a n.G.p., for the simple fact that L 1 (N) I<2 is a subalgebra of L 1 (N) Kl . On the other hand, averaging over 
K 2 gives a projection of L 1 (N) Kl onto L 1 (N) K2 . 

Therefore, every multiplicative linear functional on L 1 (N) Kl induces, by restriction, a multiplicative linear 
functional on L 1 (N) K2 . The restriction map is surjective, continuous and open from Y,(N,Ki) to H(N,K2). 

If, in addition, K\ is normal in K2, then K2 acts by composition on L 1 (N) Kl , inducing an action of the 
quotient group W = K 2 jK\ on H(N,Ki). In this case, the multiplicative functionals on L 1 (N) Kl with the 
same restriction to L 1 (N) I<2 form a W^-orbits, and therefore T,(N,K 2 ) = E(N,Ki)/W. 

This last identification can be realised on the embedded copies of the spectra as follows. 

Given D e B(N) Kl and k € K 2 , let D k e D(N) Kl be defined by D k (f ok) — (Df) o k. This gives an 
action of W on D(A^)^ 1 which respects homogeneity. 

Let V) m (N) Kl be the homogeneous component of degree m of H)(N) Kl , and choose m large enough so 
that V = E m <rh D m (N) K i generates D(N) K ^ . 

We also choose T>\ — (Di, . . . , D q ) a basis of V, which gives an embedding Ep x of E(iV, K±) in M 9 . Then 
W acts linearly on V, E-j^ is invariant under this action, and the embedding of E(AT, K%) intertwines the 
I^-actions. 

The choice of the basis V\ also introduces coordinates (£1, . . . , £ 9 ) on V. We fix a Hilbert basis p(£) = 

(Pi (&•••. Pr(0) Of V(V) W . 
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Lemma 3.1. For j = 1, . . . ,r, let D'j = pj(D\, . . . , D q ). Then T>2 — (D[, . . . , D' r ) is a system of generators 
for B(N) K ? and Sp 2 = p{T, Vl ) . 

Proof. It is quite clear that each D'. is I^-invariant, hence in D(N) K2 . Take now D £ D(N) K2 C D(7V) Kl . 
Then there is P £ V(V) such that D = P(Di, . . . , D q ). For every w £ W, we also have D = D w = 
(P ow)(D 1 ,...,D q ). IfP = /^(P oio) dm, we also have that D = P(D 1 ,...,D q ). Since P e V(V) W , it 
equals Q o p, for some Q G P(M r ). Then D = Q{D[, D' r ). This shows that V 2 generates B(N) K * . 

The equality £x> 2 = P^xO is obvious. □ 



We can give now the proof of Theorem 1.2 following the lines of [3], Section 8. 

Proof. Denote by Gi and G2 the spherical transforms of (N,Kx) and {N,K.2) respectively. Any F £ S(N) K2 
admits both transforms Q2F and GiF, and they are related by the identity 

(12) g 1 F = g 2 Fo Plsvi . 

By hypothesis, Q^F £ <S(£x>i)- Given r £ N, there are s = s(r) £ N and u r £ S{M. q ) such that 

(13) u r] =QiF, |K||( r ) < Crll-Fllw ■ 



Since P is ^-invariant, Q\F is IF-invariant, so that also u r o w satisfies (13) for every w £ W. Set 
u r = J w u r o wdw, and v r — £wu r , where £w is the linear extension operator provided by the Schwarz- 
Mather theorem (see Theorem 2.5). By (12), v r is a Schwartz extension of G2F. 

Fix now p £ N. By the continuity of £w, there is r such that ||£w<?||(p) < llff II (-r) for every 5 £ ^(IR 9 ) 1 ^. 
Therefore, 

Kllw < C' p \\F\\ is) . 

Passing to the quotient, this provides the inclusion G2 ■ S(N) K2 — > <S(Si» 2 ) and its continuity. The inverse 
inclusion and its continuity are guaranteed by Proposition |2.4| □ 



4. Rank-one nilpotent Gelfand pairs 



Many rank-one nilpotent Gelfand pairs can be obtained as a result of the following procedure. 

Definition 4.1. Let (N, K) be a nilpotent Gelfand pair and Zq C N a non-trivial /^-invariant central subgroup 
of N. Then (N/Z , K) is again a Gelfand pair, see e.g. [Mj. The passage from (N, K) to (N/Z , K) and the 
resulting pair (N/Zq, K) itself are called central reductions of (N, K). 

In this section we assume that the centre 3 of n is rank-one, i.e., that a generic if-orbit in 3 has codimension 
one. As pointed out in the Introduction, this implies that [n, tl] = 3, and hence 

n = t> 8 3 • 

Up to scalars, 3 admits a unique if-invariant inner product, which we assume to be fixed once and for all. 



4.1. Classification. 

Table [l] below is derived from Vinberg's list [571 HH] (cf. Appendix) . It gives the list of all pairs (N, K) 
where 

(i) 3 is rank-one and of dimension higher than one; 

(ii) K is minimal, in the sense that (N,Ki) is not a n.G.p. for any normal subgroup K\ of K: 

(iii) is irreducible under the action of K. 
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K 





3 


cf. Theorem 


7.5 




1 


Ui x Sp n 


C 2n 




C 


(7) after central reduction 


2 


Spi x Sp„ 


H™ 






(9) 






3 


Spin 7 


K 8 




K 7 


(2) 






4 


u 2 


C 2 




su 2 


(6) after central reduction 


5 


Sp 2 


H 2 




ffS 2 H 2 


(7) after central reduction 


6 


U 2 x SU 2 


C 2 ® 


C 2 


su 2 


(11) after central reduction 


7 


SU 2 x SU n (n > 3) 


C 2 ® 


C" 


su 2 


(11) after central reduction 


8 


S0 2 x (S Pl x SpJ 


M 2 <g> 


HP 


SPl 


(12) after central reduction 


9 


S0 2 x Spin 7 


K 2 <g> 


M 8 


R 7 


(8) 






10 


so 3 


R 3 




so 3 ^ A 2 K 3 


(1) 






11 


su 3 


C 3 




A 2 C 3 


(5) 






12 


G 2 


M 7 




R 7 


(3) 







Table 1. Pairs with dim 3 > 1, K minimal, irreducible, 3 rank-one 



In view of the results of [3] for the case where N is the Heisenberg group, and of Theorem [T2| it will be 
sufficient to consider the pairs in this list. In fact, one can check from Vinberg's list that if (N,K) is n.G.p. 
satisfying (i) and (iii) and K admits a normal subgroup K\ such that (N,Ki) is a n.G.p., then (i) and (hi) 
still hold for this pair. 

The symbol EI denotes the quaternionic algebra, O the Cayley algebra of octonions, HS 2 M 2 the space of 
quaternionic Hermitian 2x2 matrices with trace zero. 

The Lie brackets [ , ] : D x ( — > 3 and the if-actions are as follows (unless specified otherwise, vectors 
must be understood as columns). 

Line 1. [v,u] = \iJu, where J = ( ^ is the canonical symplectic form on C 2n . The factor Sp„ of K 

acts linearly on and trivially on 3; the torus Ui acts by (v, z) \ — > (e lt v, e 2lt z). 
Line 2. [w, u] = Im (v*u). For k — (ki, fc 2 ) G Spj x Sp n , k ■ (i>, z) — (fovfc*, kizk*). 

Line 3. Identify M 8 with O and R 7 with the space of imaginary octonions ImO, [v,u] — | (vu* — uv*) where 
multiplication is in O. K acts via the spin representation on O and as SO7 on ImO. 

Lines 4,6,7. Regarding the elements of as 2 x p complex matrices, v — ( Vl ) with «i,w 2 G C p , we have 

[v,u] = vu* — uv* ~ ijtr(wu* — uv*). With H denoting the second factor of K, the action of 
k = (k 1 ,k 2 ) G (S)U 2 X H is k ■ (v, z) = {kivk^.kizk*). 

Line 5. [v, u] — viu* — uiv* — |tr (viu* — uiv*). For k G Sp 2 , k ■ (v, z) — (kv, kzk*). 

Line 8,9. Regarding v as 2 x n quaternionic matrices (resp. octonionic columns), v — I 1 J , the Lie bracket 

\v 2 J 

is [v,u] = — ~(viu* — u%v* + v 2 u* 2 — U2V 2 ). The second factor of K is the same group as at line 2 
(resp. 3) and acts on HP (resp. O) in the same way, while the first factor is S0 2 acting linearly on 
the M 2 factor of and trivially on 3. 

Lines 10,11. [v, u] =t)Aii and K acts by the natural action on both and 3. 

Line 12. Identifying both copies of M 7 with ImO, [v,u] = ^(vu* — uv*), where multiplication is in O and with 
G 2 acting by automorphisms of O on both and 3. 

The table is split into three blocks: 

(i) the first block contains the pairs in which the action of K on n is "doubly transitive", cf. [17], i.e. 
the orbits are products of spheres in and 3 respectively; 



10 



(ii) the second block contains the remaining pairs in which N has square-integrable representations, 
cf. [20]; 

(iii) the third block contains the pairs in which iV does not have square-integrable representations. 
We notice that all the algebras n in the first two blocks are -ff-type. 

From Appendix, we choose a Hilbert basis p for each pair in Table [T] For pairs in the first block, we 
simply have 

P=(MM*| a ). 

For the other cases, we have 

P = 0»,Pe )3 > M 2 ) i 

natural choices of p and po.j being listed in Table [2] with the following notation: 
Lines 4-7. The inner product on is (u, u) = Retr (vu*), and similarly on 3. 

Lines 10,11. Let (ei, 62,63) be the canonical basis of over K = K, resp. C. The vectors [e2, 63], [e3,ei], [ei,e-2] 
form a basis of 3 over K. Hence we identify A 2 M 3 and A 2 C 3 with R 3 and C 3 respectively. 





Po 




4 


\v\ 2 




iv* zv 


5 


M 2 




V* zv 


6 


\v\ 2 


tr (v*v) 2 


itr (vzv*) 


7 


\v\ 2 


tr (v*v) 2 


itr (vzv*) 


8 


\v\ 2 


M 4 + 2\ Vl v*\ 2 + H 4 , \ Vl \ 2 \v 2 \ 2 - (Re(v lV *)) 2 


Re ((«i«2 - V2V*)z) 


9 


\v\ 2 


M 2 h| 2 -(Re(i^ 2 *)) 2 


Re ((vi«2 — V2V*)z) 


10 


M 2 




l vz 


11 


\v\ 2 




Re ( Sjz) , Im ( f vz) 


12 


\v\ 2 




l vz 



Table 2. 



4.2. Generalities about Ed. 

Having chosen p as above, we obtain by symmetrisation a generating family T> of D(A)- Rr consisting of: 

(1) the family C of operators corresponding to the elements of p D ; C always contains the sublaplacian 
L, coming from the symmetrisation of \v\ 2 ; it also contains one more operator at lines 6,7,9, and two 
more operators at line 8; 

(2) the family Ai of operators corresponding to the elements of p 0i3 ; for the pairs of the first block, A4 
is empty; for the pairs of the second and third block, it contains a single element M, except for the 
pair at line 11, where M. consists of two operators M\ and M2. 

(3) the central Laplacian A, corresponding to |z| 2 . 

We denote by d the total number of elements of T>. Table [2] shows that d can be equal to 3, 4 or 5. For 
the elements £ e R d we use the double notation £ = (£1, . . . and £ = {£,c,£,M,£.a)- Then Dd = A, and 
we assume that Di = L. 

We split S(A, K) as S(A, AT) rc suS(A, AT) sin s, where the singular part E(iV, AQ" ing consists of the bounded 
spherical functions that do not depend on the 3-variable. In view of Theorem 1 2. l[ E(iV, K) slng contains the 
bounded spherical functions associated to one-dimensional representations of N. Alternatively, one can say 
that E(iV, K) YCg contains the bounded spherical functions associated to coadjoint orbits of maximal dimen- 
sion. We also say that a bounded spherical function is regular, resp. singular, if it belongs to E(A, AT) rcg , 
resp. E(N,K) siDg . 
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In our context, transporting the splitting from S(iV, K) to £x>, we obtain: 

(14) S^ ng = U e S„ : Ca = 0} = {£ G : = U = 0} . 

Each operator ZX,, j = l,...,d is homogeneous with degree denoted by jj. The sublaplacian L is 
homogeneous of degree 71 = 2. All the other operators of £, when present, are homogeneous of degree 4. 
The elements of M. are homogeneous of degree 4 for the pairs in the second block, and of degree 3 for the 
pairs in the third block. Finally, A is homogeneous of degree 4. 

Then S-p is invariant under the dilations (£1, . . . , ^) 1 — > (t 7l £i, . . . , i 7d ) for t > 0. 

Since L is hypoelliptic, the subelliptic estimate H-D7/H2 < Cj\\L lj / 2 f\\2, for / € S(N), implies that the 
coordinates £j, j = 2, . . . , d of the points in Ep satisfy the following inequalities: 

(15) &l<G;lfcl*. 
On S-p we then have 

d 

(16) l£|=£l&#=6- 



4.3. Quotient groups. 

Let Co G 3 be a unit vector and let K' be the stabiliser of MCo m K. Then K' = K' xi Z2, where ifg is the 
stabiliser of Co m K. Set also AT' = N/ exp(C c |-), where the orthogonal complement is taken in 3. 
By [5], (iV, if ) is a n.G.p., that we want to describe case by case. 

4.3.1. The pairs (N' , K' ), first and second block. In this case n is an iZ-type algebra. More precisely, admits 
an inner product such that (10) holds. Then the skew-symmetric form uiq {v,u) — ([v,w],Co) = (Jco v ' u ) on 
t> is nondegenerate, and N' is isomorphic to the Hciscnbcrg group H s , with 2s = dimO. 

Since J^ is both skew-symmetric and orthogonal, it satisfies the identity J 2 o — —I, i.e. it defines a 
complex structure on 0. 

It follows from (10) that K' consists of the elements of K commuting with J^ . Therefore, K' acts by 
complex linear transformations on (t>, J( ), and under the identification N' = H Sl Kq is identified with a 



subgroup of the unitary group \J S . We keep the notation introduced in Subsection 4.1 

• First block 

Lines 1,2. It is easy to verify that Kq = Sp„ at line 1, and Kq — Ui x Sp„ at line 2. 

Line 3. The stabiliser of Co € ImO is isomorphic to Spin 6 = SU4. Since it acts nontrivially on (t>, J^ ) 
as a subgroup of U4, this can only be SU4. 

• Second block: we write as the sum t>i © t>2 with rq = | ^ q 1 ^ j"' °2 = | J j ■ With the following 

choice of Co> for lines 4-6, it is easily checked that tq and O2 are two equivalent irreducible subspaces 
of D under the action of K' and J^ a v = (ov. 

Lines 4,5. Taking Co = ^ for line 4 and Co = ^ J for line 5, then K' Q is Ui x Ui and Sp x x Sp x 

respectively, where the two copies of Ui or Sp 1 act independently on the two subspaces tq and 
02- 

Lines 6,7. Taking Co = ( q ^ j , then K is (S)(U 1 x Ui) x SU P where SU P acts linearly on tq, X>2- 



Lines 8,9. We choose the same Co as at line 2 (resp. 3). The stabiliser of Co is SO2 x H' , where H' is the 
stabiliser of Co for the second factor of K; hence H' is Ui x Sp n (resp. Spin 6 ). The complex 
structure J^ allows us to identify tq and t) 2 with the same copy of C 2n (resp. C 4 ). The group 
H' acts on each of tq, X>2 like Ui x Sp n on C 2 ™ (resp. SU4 on C 4 ), with the elements of SO2 = Ui 
acting as intertwining operators. We denote the Hermitian product on C 2 ™ (resp. C 4 ) as ( , ). 
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4.3.2. The pairs (N',Kq), third block. In the other cases, the form ujq has a nontrivial radical, which has 
dimension r — 1 for the groups at lines 10 and 11, and dimension r = 2 for the pair at line 11. Then N' is 
isomorphic to the direct product of i?(dims-r)/2 with R r . 

Lines 10,11. We choose Co = Thus Kq is isomorphic to SO2, resp SU2, and, under the action of K' , splits 
as the sum of two irreducible subspaces Oi — Ke x Ke2 and 2 = Ke 3 . 

Line 12. Let (ei, . . . , ei) be the canonical basis of — ImO. We choose (q = e-j. Then splits under K as 
the sum of Ox = Rex ffi . . . ® M.ee and O2 = K.e7. Left multiplication by e 7 gives a complex structure 
on Ox 1 which can then be identified with C 3 , with K Q acting on it as SU3. 

4.3.3. Hilbert bases. Table [3] summarises the result of the two previous subsubsections. For each pair 
(N',K' Q ), it also gives a Hilbert basis p' D for the action of K' on 0. A Hilbert basis p' n , for the action 
of K on n' is obtained by adding the coordinate function on R£o- 





N' 


K 




1 


H 2n 


Sp„ 


\v 


2 




2 


H 2n 


Ux x Sp n 


V 


2 




3 


Hi 


SU 4 


\v 


2 




4 


H 2 


Ui x Ui 


\v 


2 




5 


Hi 


Spi x Spx 


V 


2 


Kl 2 -hl 2 


6 


Hi 


Ux x Ux x SU 2 


V 


2 


tr(v*v) 2 , \ Vl \ 2 ~\v 2 \ 2 


7 


H 2n («>3) 


S(Ui x Ux) x SU„ 


V 


2 


tr(v*v) 2 , |wx 2 ~ M 2 


8 


Hi n 


Ux x (Ux x S Pn ) 


V 


2 


| W i| 4 + 2| W x W *| 2 + |x;2| 4 , |«x| 2 |« 2 | 2 - (Re K,v 2 )) 2 , -21m (v u v 2 ) 


9 


H s 


Ux x SU 4 


V 


2 


\vi\ 2 \v 2 \ 2 -(Re( Vl ,v 2 )f , -2Im( Vll v 2 ) 


10 


Hi x R Va 


Ux 


\v 


2 


v 3 


11 


H 2 x C„ 3 


su 2 


V 


2 


Re V3 , Im W3 


12 


H3 X Ry 7 


su 3 


V 


2 


w 7 



Table 3. The quotient groups TV', with K' Q and p'„ 



4.3.4. Differential operators. In each of the cases under consideration, we take the generating system T> = 
(D[, . . . , D' d ) of V)(N') K ° obtained by symmetrisation of the Hilbert basis p' formed by the invariants in p' v 
in Table [3j with the addition of the coordinate function on K^o ■ Notice that the cardinality of p' is the same 
cardinality d of the corresponding Hilbert basis p on n. 

As for the operators of V, each operator DL j = 1, . . . , d is homogeneous with degree denoted by 7^ and 
£>x is hypoelliptic and homogeneous of degree j[ = 2. The coordinates rjj, j = 1, . . . , d of the points in 
satisfy the following inequalities: 

(17) l»&l<q;-M? 



4.4. Radon transforms and structure of Sp S . 

The notation used throughout assumes that N and N' arc identified with their Lie algebras via exponential 
coordinates. By £o" we denote the orthogonal of ( in 3. 

Following |llj . for any integrable i^-invariant function F on N, we set: 

TZF(v, t) = [ F(v, Ko + C)dC , («, t) e N' , 
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This defines a function 1ZF € L 1 (N') K , that we call the Radon transform of F (more precisely, it encodes 
all the Radon transforms, in the ordinary sense, of the functions F(v, •) on 3). The operator 1Z is linear, K'- 
equivariant and respects convolution. Furthermore it maps continuously L 1 (N) K to L 1 (N') K , and S(N) K 
to S(N') K '. 

We extend the definition of K to B(N) K in the following way: if D £ D(N) K , then we define D' = KD e 
D(/V') X ' by 

{D'G)oq = D(Goq), GeC™(N'), 
where q : N — > N' is the quotient mapping. Easy changes of variables show: 

Lemma 4.2. If D € V)(N) K is the symmetrisation of the polynomial P on n, then D' = 1ZD is the 
symmetrisation of the restriction P\ , of P to n'. 

We recall that, by [S], (N'K') and (N',K' ) are n.G.p. We say that a bounded spherical function for 
(N',K') or (N',K' ) is regular if it is associated to an infinite-dimensional representation of TV'. 

By the rank-one action of if on 3, every irreducible unitary representation of N factors through a repre- 
sentation of a conjugate of N' under K. Via Theorem 2.1 this implies that every bounded spherical function 
for (N, K) can be expressed as 

(18) ift(v,z) = [ cp(kv,(kz, Co)) dk , 

Jk 

and that this is a 1-to-l correspondence between Y,(N',K'y cg and E(N, J ftT) rcg . 

It is convenient to regard 1Z as an operator from 71 : L 1 (N) K to L 1 (N') K o, disregarding the extra 
invariance under Z2 at this stage. 

A ifg-bounded spherical function on N' has the form 

(19) ij(v,t) = Mv)e lM , 

with A ^ if and only if ip is regular. Choose w € K' \ K' where w(v, t) — (Wq(v), —t) . Then also 

i) w (v, t) = $ o w(v, t) = (ip o w ){v)e~ tM 
is a i^Q-bounded spherical function and the map 

is a 2-to-l correspondence between E(7V',^) reg and Y,(N\K'Y^ (cf. Section^. 



Denote by £(TV', i^) rc s<+ the subset of £(7V',i^) rc s consisting of the functions ([19]) with A > 0. Then 
E(N', K' o y^'+ is open in E(7V', K' ) and S(JV', ^) reg is the disjoint union of S(JV', ^) reg '+ and wT,(N', K^ IC ^+ . 

We denote by G'o the Gelfand transform for (N',K' Q ). The following statement summarises our previous 
remarks. 



Proposition 4.3. If tp is a bounded spherical function in (N',K' ), then (18) defines a bounded spherical 
function V> B for (N, K). For F G L l (N) K , 

(20) / F(v,z)ip i (-v,-z)dvdz= / nF(v,t)i/}(-v,-t) dvdt , 

Jn JN' 
i.e. ! GF{^) = G' Q {llF){i}). 

For any D € D(N) K , TZD and D share the same eigenvalue on ip and respectively. In particular, ijfl 
is regular if and only if if) is regular. 

Moreover, (4> w f = ^ and the map ip 1 — > 1$ is a bijection from £(/V', i^) rc s>+ onto Y,(N,Ky es . 

Comparing the restriction to n' of the elements of p with the elements of p' , cf. Tables [2] and [3j we see 
that every element of p' is the restriction to n' of an element of p. If T>, V are the corresponding systems of 
differential operators, it follows that V = TZV. 

In the next statement the map 8 is defined so that 9 _1 is the same as the map ip H> ip* in (18 1 for the 
embedded copies of the spectra in R d . The notation is set so that S-p = E(7V, K) is in the £-space, while 
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T>d< — T,(N',Kq) is in the 77-space. The coordinates £1,771 represent the eigenvalues of the sublaplacians on 
N,N' resp., while £<j, resp. 77^, represent the eigenvalues of the central Laplacian on N, resp. the derivative 
in the direction of MCo ■ 

Lemma 4.4. Le£ : x M\{0} -> M d &e given &y ©(£1, • • • > = (»7i> • • • i %)> w/iere 771 = fi, r? d = Vld 

m a/Z cases, and: 

_ 1 



(21) 



172 


= 6C3 3 






at lines 4,5,10,11,12 


(d = 


3) 


172 


= 6 






at lines 6, 7, 9 


(d = 


4) 


172 


= 6 




= & 


_ 1 

i]4 = £ 3 £ 5 2 at line 8 


(d = 


5) 


172 


= 6£ 4 " 4 






at line 11 


(d = 


4) 



T/ien maps homeomorphically onto £p, S:+ . 



5. A Geller-type formula with remainder term 



We begin now the proof of Theorem |1.1| As we have poined out in the Introduction, it suffices to prove 
([3]). We have also commented that we treat all pairs in Tabl e [T] in a unified way. For the pairs in the first 
block a separate simpler proof is possible, based on Theorem |1.2| (but with a variant for the pair at line 3). 

Due to the absence of mixed invariants, some of the statements below are either vacuous or considerably 
simpler to prove if specialised to pairs in the first block. 



Proposition 5.1 below is our key tool. It is a (less refined) analogue of Theorem 2.2 in [2], Proposition 
7.2 in [3] and Proposition 5.2 in [TT]. The structure of the result is derived from 13J. It hints that, for 
F e S(N) K , QF has some kind of Taylor development at s . 

Proposition 5.1. Let (N,K) be one of the pairs of Table [7J Given F G S(N) K , there exist functions 
g jk e S{R d "), j e N* M , k e N, such that, for every n G N, 

(22) F= -^MJA k g ]k (£)6 + Rn , 

K 



with R n e S(N) K , 



(23) R n = J2 d * R '* > 

\a\=n+l 

with R' a € S{N). Moreover, each g.jk and R a depends linearly and continuously on F. 



Notice that for the pairs in the first block, M. is empty, so that (22 1 only contains the sum over k. 



5.1. Tools for the proof of Proposition [57T] 

Let us set first some notation for the pairs in the second and third block of Table [T] For each n £ N, we 
denote by E n the set consisting of the following pairs: 



(24) 



(2m + 1, 0), (2m -1,1),..., (1, m), (0, m + 1) if n = 2m 
(2m + 2,0), (2m, 1), . . . , (0, m + 1) if n = 2m + 1 , 



in all cases but line 11, where we define E n as the set of triples (j, k) = (ji, J2, k) £ N 3 such that k) runs 
over the set given in ( 24 1 . 
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Lemma 5.2. Let n e N fee fixed. With denoting elements of x 3, suppose that £) G <5(n) A 

vanishes at £ = im'f/i derivatives in £ up to order n. Then 

(25) «(w,C)= E C) J IC| 2fc ^fc (p(o,, C)) 

(J.fe)e-En 

where the hjk G <S(n) depend linearly and continuously on u. 
For a pair in the first block, (251 reduces to 

u{ujX)^\C\ 2[n,2]+1 K{ujX) , 

wii/i /i„ G S(n) K depend linearly and continuously on u. 



The proof of Lemma |5.2| is based on the following two simple lemmas, the first of them being an adapted 
version of Hadamard's lemma. 

Lemma 5.3. Let h G <S(K™ x M™). If h vanishes at each point (x, 0) with all derivatives up to order k, then 
there exist h a G 5(M n x R m ), a G N m , |a| = k + 1, depending linearly and continuously on h, such that 



h(x,y) = ^2y a h a (x,y) 



Sketchy proof. It suffices to consider the case k — 0. Fixing ip G C^°(M. m ) and ip = 1 on a neighborhood of 
the origin, we have: 

h(x,y) = / d r h(x,ry)dr 



= y2yAi J (y) / (d yj h)(x,ry)dr + y 3 h(x,y) — 
3=1 V Jo 



■<Kv) \ 



\y\ 2 J 



□ 



Lemma 5.4. 1. Let a G M m \{0}, m > 2. The polynomials (a,r) J , j G N, as functions of t in the unit 
sphere S" 1 ^ 1 G 3, are linearly independent. 
2. Let a G C m \{0}, m > 2. TTie polynomials (a | r)-?' 1 (a | t)^ , j G N 2 , as functions of t G are 
linearly independent. 

Sketchy proof. In both cases we can assume that a = (1, 0, . . . , 0). In the real case, if P(t) = X)j l =o c j( a > T )° 

is zero on the unit sphere, then J2^=o c j T i * s zer0 f° r T i *= [ — -*■> -*-]■ The same holds in the complex case, with 
7~i in the unit disk in C. □ 

Proof of Lemma \5.S\ We begin with a pair in the second or third block of Table [T] except for the one at line 
If. 

In this case there is only one mixed operator p D . 3 which can be written as: 

m 

(26) Pt>,i(u, C) = = X] C^( w ) 
where to = dim 3. 

Consider first the case n — 0. Then it is if-invariant and u(uj, 0) = for every oj. Then, with h = £r-u, 

u(u>,Q = h(p(u>,Q) ■ 

It follows that /i(£c,0, 0) = whenever £c G /?»(&)■ Replacing ft by /i(£) — ft(£c, 0, O) i 0(^m ) £a), with 
if) G C^°(K 2 ) and tp = 1 on a neighborhood of the origin, we may assume that /i(£c, 0,0) = for every 
(6c, 0,0) GM d . 

By Lemma [B~3l 

M£)=6vM£) + £^2(£) , 

with hi,h 2 G <S(R rf+2 ) depending linearly and continuously on h, hence on u. Then 

(27) u(u J ,0=p(uX)h 1 (p(u J ,()) + \<;\ 2 h2(p(0JX)) ■ 
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This gives (25) for n = 0. 

Suppose now that (25) holds for n, and suppose that u vanishes at Q = with all derivatives in £ up to 
order n + 1 . 

We differentiate ( 25 ) n + 1 times along a given direction r in 3 at £ = and impose that the result is 
zero. By the iif-invariance of each term in (25), we can assume that r — ei, i.e., we differentiate in £1. Each 
term p(w, () J \C\ 2k appearing in (25) is homogeneous of degree n + 1 in £, except for the single term |£| 2 ( m+1 ) 
when n = 2m, which is homogeneous of degree n + 2. Therefore, 

9£ +1 u(a,,0)= ]T ^ +1 (p( W ,C) J IC| 2fc )(^0)^(p( W ,0)) . 

j+2fc=n+l 

For j + 2fc = n + 1 we have 



^>(^0->ici 2fc ) = 



.7 



j!pi(a;)'(2fc)! = (n + l)!pi(w) J 



Therefore, for every w £ c. 

£ pi(w)^(p.M,0,0)=0. 

j'+2fe=n+l 

Noticing that p(w,£) = pi(w), by if-invariance we have 

p(w,T)^ ifc (p B (w),Q l 0)=0, 

j+2fe=n+l 

for every t (£ S } and every wee. Fix w G and set p(ui,t) — (a,r), with a = . . . ,p m (w)J € K r 

Then a is generically non-zero and, by Lemma 5.4 part 1, hjk{po{w), 0,0) =0 for any (j, k) with j + 2k 
n + 1. 

If n is odd, the pairs (j, k) with j + 2k = n + 1 exhaust £J n . Then, applying (|27|), 



u( W ,o= £ p(^,o-'ici 2fe (^,c)^ fe (^,c)) + ici 2 ^ fe (p(^c))) , 

j+2fc=n+l 

where each term contains a factor pfw,^)-? \C\ 2k with (j', fc') g E n +i- 

For n even the argument is the same, once we have noticed that the pair (0, m + 1) belongs to both i?2m 
and E 2m+1 . 

Consider now the pair at line 11 in Table [T] In this case let qi, q 2 be the two elements of p D 3 in Table [2] 
and set: 

3 

p(<*>,0 = <?i(^,C) +«92(w,C) = X^MG • 



Then (25) is equivalent to: 

(28) «(w,o= E p^.o^pO^'ici^fcCpC^o) 

(j,fe)e-E„ 

where the hjk € <S(tt) are new functions depending linearly and continuously on u. 
The case n = is a simple adaptation of the proof for the other cases. 

Suppose now that (25) holds for n, and suppose that u vanishes at £ = with all derivatives in Q up to 
order n + 1. By the inductive assumption, we know that (25) and thus (28) hold. 

In this proof we denote by d T and d T the holomorphic and antiholomorphic derivatives in a given direction 



t in 3 



We impose the condition d r T x d r2 u(w , 0) = 0, for n + ri = n + 1. 



By if-invariance, we can assume that t = e\. We have 



r 1 \r 2 \pi{u) :il pi{uj) :i2 if ji = r x - k , j 2 = r 2 - k 
otherwise. 
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Arguing as in the previous proof, this implies that for every n, > k with n + r2 = n + 1, 

^ pi (c)^pTH 32 fejfc (fh (w), 0, 0) = , 

|j|+2fc=n+l , jx— 33=ri— r 2 

By if-invariance, for the same values of r 1; r 2 , 

J2 P(w, Tp fc ifc (p, (w) , 0, 0) - , 

| j|+2fc=n+l , ji— J2=rx— r 2 



for every t € S and every w£t). The proof continues as in the previous case, using part 2 of Lemma 5.4 



Finally, the case of a pair in the first block of Table [T] is much simpler to prove, following the same 
lines. □ 



5.2. Proof of Proposition pTT 



We assume that (N, K) is a pair in the second or third block of Table [TJ The remaining cases are simpler 
and the modifications to the proof below are left to the reader. 

Notice that the spherical functions in E^" 8 coincide with the spherical functions for the abelian pair 
(t>,K). Moreover, for F G V-{N) K and & = PoM, wet), 



(29) 



SF(6c, 0,0) =F(w,0) 



where F is the Fourier transform of F on © 3 (identified with 0* 3* v ia the if-invariant scalar products). 
Let u(v) = J F(v, z) dz G 5(o) A ', and .900 = £ku. By Proposition^ F 00 = g 00 (C)S € S(N) K . By 



eF 00 (£ £ , 0, 0) = ff0 o(^) = £F(£ £ , 0, 0) 



for every £c G Po(o). In particular, 



Fqq(v, z) dz = 



Let Rq — F — Fqq. Since / F(v, z) dz = for every v, Lemma 5.3 implies that, with m = dimj, 

n 1 

with i?^- G S(N) depending linearly and continuously on Ro thus F. 
This proves the statement for n = 0. 



Suppose now that the statement is true for n, and consider the function _R„ in (22 1. By (23), i?„(w, £) G 



<S(ti)' K ' vanishes with all derivatives up to order n for £ = G 3. By Lemma 5.2 

i4(c,o= 53 p D >,o j icrM/^c)) , 

U,k)eE n 

where the hjk G ^(n)^ depend linearly and continuously on R n , thus on F. Undoing the Fourier transform, 



(30) 



R n {v,z)= 53 Po jr 1 V ,t- 1 V i yA k F jk (v,z) 



where the Fjk are in S(N) , depend linearly and continuously on R n , and hence on F. 
We replace now the constant coefficient operator p(i _1 V , i _1 V 3 ) by D using: 

Lemma 5.5. For j G N, or j G N 2 in the case of line 11, 

^'-^(rV,,!- 1 ^)^ 53 a z Q p Q ( v ,v D ,v 3 ) 

i«i=bi+i 

where P a (v, V D , V 3 ) is a differential operator in D and 3 wift polynomial coefficients in v. 
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Proof. To fix the notation, we disregard the case of line 11. The proof for that case requires only minor 



modifications. Fix an orthonormal coordinate system (vi, . . . , v q ) on t>. By (261, 



p(r 1 v„,r 1 v,) = 53w(a Ol ,...,a o ,)0* , D = J2\'{ Pi )(v 1 ,...,v q )d zt , 

e=i £=i 



where A' is the modified symmetrisation operator in ^ and 

m 

V r = d Vr + E Xre(v)d ze , 

the Xri being linear functionals. 

Each pi is a quadratic form on for the pairs in the second block, and linear for the pairs in the third 
block. We restrict ourselves to pairs in the second block, the other cases being even simpler. 

With these assumptions, D is a sum of terms, each being a product V r V s d ze . We have 

mm m m 

V r V s = d Vr d Vs + ^2c rsk d Zk +^2xsk(v)d Vr d Zk + ^Xrkiy)d Vs d Zk + E Xr^Mfet")^^ . 

k=l k=l k=l ki,ka=l 

So, there are polynomials Qj such that 

771 

V r V s d Ze = d Vr d Vs d ze +^2d ze d Zk Q rsk {v,\7 V7 S7 } ) . 

k=l 

Summing over all terms, we have that 

m 

D = P (i- 1 v ,r 1 v i )+ E 9 Zfe ^P M («,V D ,V 3 ) . 

k,e=i 

It is then sufficient to use this expression of D recursively. □ 



Using Lemma 5.5 we have: 



R n = £ -L^A*F ifc + E »■ 

j+2fe=n+l ' |a|=n+2 

Observing that each term in D 3 A contains n + 1 derivatives and using the case n = for the Fj k' 

m 

Fjk = 9jk{C)S + E dz e Ri k , 
e=i 

we have: 

Rn= E lL Dj ^9iMh+ E 

j+2k=n+l^' ' \a\=n+2 

with each gj k £ and R a £ S(N) depending linearly and continuously on F. 

This finishes the proof of Proposition |5.1| 



6. Proof of Theorem 11.11 



We denote by g?„ the number of invariants in p v , which is also the cardinality of C Then d — d + 2 for 
all cases except for line 11, where d = d„ + 3. 

For pairs in the first block of Table [T] the variable £_A4 must be disregarded. 
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Let F e S(N) K . Take the functions g jk G S(R d ), and R n of Proposition [El] Let tjj e S(R d ) be any 
function such that: 

(31) ^^(^,0,0)=.^) .. 

Fix <p(£.Mj £a) smooth, with compact support and equal to 1 on a neighborhood of the origin. Then iptp G 
S(R d ). 

Dehne G G S{N) K as 

(32) G = F-(^)(C,M,A)S Q . 

We prove that from any such ip we can construct a Schwartz extension of the Gelfand transform QG of G. 
Lemma 6.1. For every n, G = Xj a |=n ^z^a with H a G S{N), \a\ = n. 
Proof. Fix n G N. Let 

T «®= E %|r^)- 

\j\+2k<n J ' 

Following [3], Proposition 7.2, we modify T n outside of Ex> so to obtain a Schwartz function. More 
precisely, using (15 1, we define Hi, resp. f2i, as the set of points £ G M d such that, for I = 2,...,d, 
161 < 2C/(1 + 2 , resp. < 4Cj(l + |^i|)"^, and S! 2 , resp. f2 2 , as the tubular sets of points £ G R d such 

that Ya=2 €f - 1 j res P- Ef=2 < 2 - We set ^ = ^i u ^2, & = f2i U f) 2 . 

Fix x € C°°(]R' J ) such that x = 1 on SI, x = on the complementary of (l; as the distance between 
and the complementary of is positive, we can assume that each derivative of \ is bounded. Then xT n is 
a Schwartz function on K d , which coincides with T n on a neighborhood of E-p. 

We set: 

M0 = m-x(0T n (0 , t^ n (0 = (i-^,€A))x(Orn(0-V'n«M^,€A) • 

t/>„ and ip n are Schwartz functions of M. d . As 0„ coincides with T n — -0 on a neighbourhood of (£c, 0, ) G 



5.3 



ip n vanishes with all derivatives in £m,£,a up to order n at each point (££,0,0) G K d . Using Lemma 

M0= E ^^(6. 

b'|+fc=n+l 

with ^j fc G 5(M d ). 

Furthermore, for £ G E-p we have: 

QF(0 = GG(0 + (V^)(0 = T n (0 + GR n (0 , 

therefore: 

Inverting the Gelfand transform, 

G = R n + ]T M j A% k (C,M,A)S , 

|j|+fc=n+l 

where each term ./VP A fc contains at least n + 1 derivatives in the 3-variables. □ 

Denote by G the Radon transform of G on N' . Assume that the derivatives d" are referred to an 
orthonormal frame in 3 having £ as its last element. For every n, 

G(v,t) = dl l H n (v,t) , 

where H n is the Radon transform of Hq ,...o,n given by Lemma 6.1 
This implies that Q' Q G vanishes of infinite order at r/d = 0. 

Lemma 6.2. If G is as above, there exists a Schwartz extension g of G' G satisfying: 
(1) all derivatives of g vanish at all points ofM. d with n^ = 0. 
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(2) the support of g is contained in a set where \rjj\ < C\t]i\ 2 , j — 2, . . . , d, 

(3) for every p > 0, there exist a constant C p and an integer q such that 

\\9k P) < C p \\G\\ (q) . 

Proof. For pairs in the first and second block of Table [l] each group N' is isomorphic to the Heisenberg group 
H m and Proposition 7.5 of [3] applies: there exists a linear and continuous operator producing a Schwartz 
extension go of Q' G such that all of its derivatives vanish at all points of R d with r/d = 0. 

For the third block, each group N' is isomorphic to the direct product of a Heisenberg group with an 
abelian factor. Under these circumstances, the same conclusion can be drawn from a slight modifications of 
the arguments in Proposition 7.5 of [3J. 

Using (17) we define W and f2' as the sets of points (772,...,%) £ R d_1 such that \r)j\ < 2Cj and 
\Vj\ < 4CJ-) j = 2, . . . ,d, respectively. We fix a function x' G C° c (R d ^ 1 ) satisfying x' = 1 on fi' and x' = 
on the complementary of il' . We set: 



0(77) = <^ 9a(v)x'{V2Vi 




.Mi 



if 771 = 



It is straightforward to show that g satisfies the property (1), (2) and (3) of Lemma 6.2 



□ 



Lemma 6.3. // G is as above, QG admits a Schwartz extension h such that for every p > 0, there are a 
constant C p and an integer q for which: 

(33) \\h\\ {p) <C p \\G\\ {q) . 

Then, for every 77 £ R d , a £ N d , a, r £ N, there is q = q(a, a, r) such that, 



Proof. Let g be as in Lemma 
for every 77 £ R, 



6.2 



(34) 

Using Proposition 
Then the function 



4.3 



and Lemma 



|^(77)|<C QffT ||. 9 || (9) ^L. 
Q' G and QG are related by 



if u > . 

otherwise. 



coincides with QG on Ep- 

Using (34) and the fact that, in all cases, 77 = 0(0 satisfies r]j = w ith aj £ {0, —5} , it is easy to 

verify that, for every multi-index a and every a, r £ N, there is = (/'(a, <r, t) such that 

\u\ 



\d a h{i)\<C aaT \\g\\ w)J ^^ T 



for ^ > 0. This shows that h £ S(R d ) and that it depends linearly and continuously on g. By (3) of 
Lemma 6.2 and the continuity of the Radon transform, we obtain (33). □ 



We a re now ready to prove Given p > 0, take q > p large enough so that (33) holds. By Proposi 



tion 
that 

(35) 



2.4 



for any multiplier m £ <S(R ), the function m(C,A4, A)So is in S(N) and there exists r > p such 

\m(£,M,A)5 \\ (q) < C q \\m\\ (r) . 

We observe now that, following the same lines as in [3], Proposition 7.4, the function ip can be chosen so 
that it satisfies (31) and 



|(r) < CrljFI^) 
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for some s G N. By (|32j) and (|35j), 

< \\F\\ {9) + C q \\i/><l>\\ ir) <C , q \\F\\ (ll , ) 



with q' = max(q r , s). By Lemma |6.3| 

\\h\\ {p) < C p \\G\\ {q) < C' p \\F\\ w) . 
Therefore u — h + ipip is a Schwartz extension of QF, and 

\\u\\ (p) <\\h\\ (p) +C P U\\^<C%\\F\\ {ql) 
This concludes the proof of Theorem |1.1| 



7. Appendix. Explicit invariants for pairs in Vinberg's list 

Assume that (A, A) is a nilpotent Gelfand pair and that A is exactly step-two i.e., non-commutative. As 
before, we decompose the Lie algebra n = Lie A as a direct A"-invariant sum n = ® 3, where 3 is the centre 
of n. We assume that 

(i) [n,n] =3, 

(ii) the action of A on t> is irreducible, 
(hi) dim 3 > 1. 

Then the pair (A, n) is contained in Vinberg' table, see [301 Table 3], which has 12 items. Our main goal is 
to describe the algebra IRfn]^ of A-invariant polynomials on n. 

Lemma 7.1. Let W be a finite- dimensional representation of A defined overM.. Let W(C), A(C) denote 
the complexifications ofW and A respectively. Then C[VK(C)] K(C) = R[W] K <g> R C. 

Proof. The statement easily follows from the fact that the action of A on W®rC commutes with the complex 
structure. □ 

We say that some property holds "for a generic point" , if it holds for all points of some non-empty open 
subset. Over an algebraically closed field, in our case C, we will be using a simplified version of the Igusa 
lemma, see e.g. j2H Theorem 4.12]. 

Lemma 7.2. Let G be a complex algebraic group and V a vector space of its finite- dimensional linear 
representation. Suppose that there are algebraically independent polynomial functions Fx, . . . , F rn G C[V] G 
such that 

(i) for generic point y = (yx, ■ • ■ , y m ) G C m , the level subvariety (p~ 1 {y) := {x G V | Fi(x) = yi for i — 
1, . . . , m} contains a unique open G -orbit; 

(ii) the set of points y G C m such that (p~ 1 {y) ^ contains a big open subset U, i.e., such that 
dim(C m \U) < to -2. 

ThenC[V] G = C[Fx,...,F m ]. 

Remark 1 . Usually conditions of the Igusa lemma are stated in terms of a morphism ip : V — > C m defined by 
(f(v) :— (Fx(v), . . . ,F m (v)). The level subvariety v ?_1 (y) is a fibre of this morphism. Condition ip^ 1 (y) 7^ 
means that y lies in the image of if. There is also a third condition, which is automatically satisfied in our 
case. 



Remark 2. Suppose we are in the setting of Lemma 7.1 Since all orbits of the compact group A are closed, 
generic A(C)-orbits on V — W(C) are closed as well and are separated by the polynomial invariants. This 
means that condition (i) of Lemma 7.2 is satisfied if and only if for generic y G C m the level subvariety 
(p^ 1 {y) CVisa A(C)-orbit. 

Definition 7.3. A finitely generated commutative associative algebra A is said to be free if it admits a system 
of algebraically independent generators. Such a system is called a free generating set. 
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Suppose that (N, K) is a nilpotent Gelfand pair, where n = © R is a Heisenberg Lie algebra. Then 
V := t>(C) decomposes as a sum of two complex i4T(C)-representations, more precisely V = W © W* . The 
group K(C) acts on C[W] and this action is multiplicity free, see 0J. We can express this statement as 

C [W}= V X , 
\eA(w) 

where V\ is an irreducible representation with the highest weight A and A(W) is the monoid of all appearing 
highest weights. For more details see [5] or [15]. The polynomial if (C)-invariants on V are described as 
follows C[V] K( ^ = (Vx ® V^) X(C) , where each subspace (V x ® ) K ( C ) is one-dimensional. It is know 

AeA(W) 

that the monoid A(14 7 ) is free and its generators are given in [5J or [T5J. If A l7 A2, . . . , A r are free generators of 
A(W), then Fi := (Vx^V^^ (with i = 1, 2, . . . , r) are algebraically independent generators of C[F] K ( C ). 

Other rich sources of invariants are papers of G. Schwarz [25J and by O.M. Adamovich and E.O. Golovina [TJ, 
where classification of the representations of complex simple algebraic groups with free algebras of invariants 
is carried out. In those papers the generating invariants are described in terms of some representations 
and their highest weights, which may not be very explicit. On the other hand, the degrees of generating 
invariants are also given and this can be enough in many cases in view of the following fact. 

Proposition 7.4. Let A be an algebra of real or complex polynomials equipped with an algebraic grade- 
preserving action of some algebraic group G. Suppose that A is generated by homogeneous algebraically 
independent polynomilas ai, . . . , a m and that b\, . . . ,b m £ A G are algebraically independent with degfe; = 
deg «j . Then A is generated by b\ , . . . , b m . 

Proof. The explanation is that A G and its subalgebra B, generated by bi, have the same Poincare series. 
To make it more explicit, let us compute the dimension of a linear space A G of the G-invariant polynomi- 
als of degree d. Since et^ are algebraically independent generators, it is equal to the number of m-tuples 

m 

(ti, . . . ,t m ) £ Z™ such that ^dega^ = d. Note that dim I? I~l Ad is the same. □ 

~ i=i 

In practice it is not very difficult to verify that some polynomials are algebraically independent. Usually 
we do it by restricting them to a suitable subspace. 

Trivial if-representations appearing in 3 correspond to linear if-invariants and they are often omitted from 
consideration. Suppose that ro C n is a if-invariant subspace. Then K[n] = S(n) has a if-invariant bi-grading 
with respect to a if-invariant decomposition n = Id © m, i.e., the grading components are S p (ti))S q (m). We 
have K.[ro] C R[n] and for any generating set {F\, . . . , F m } in M[n] A ', the homogeneous components of 
the Fi's of the bi-degrees (degFj, 0) form a set of generators in R.[tt>] . For that reason we do not state the 
answer for possible central reductions of n, i.e., for pairs (K, n/30), where 30 C 3 is a if-invariant subspace. 
(Note that the representation of K on n/30 is isomorphic to the representation of K on the if-invariant 
complement of 30 in n.) 

For a skew-symmetric matrix x, let Pf(x) denote the Pfaffian of x. Having an nxn skew-symmetric matrix 
z and an n- vector v, we build a new skew-symmetric matrix 




We let z denote an element of 3 and v an element of 0. The symbols \v\ and \z\ stand for the norms on 
and 3. Set v* :— tJ, where ~ stands either for complex (or quaternionic, in cases 7 and 10) conjugation. 
Our notation in case 12 needs an explanation. Here = C 2 <g>cHI n . Each vector v £ V can be presented 
as v — Wi®v\ + W2®V2, where w\,W2 £ C 2 are column vectors and V\,V2 £ HP are row vectors. Set 
v* := fei^^i + t H)2® t V2- Then vv* can be identified with a 2x2 quaternionic matrix. In this terms a 
if -invariant real- valued scalar product on t> can be written as tr (vv*) or as tr (y*v). 

The cases are numbered according to the lines of Vinberg's table, [30l Table 3]. We only unify items 4 
and 5. 
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Theorem 7.5. Generators Fj G B^n]^ can be chosen as follows. 
K = SO„, n = R n © so„, n = 2? : 

: tr (z 2fc ) , 1 < k < I - 1 , Pf (z) , W 2fe « , < k < I - 1 ; 
F = SO„,n = R"©so„,n = 2Z + l : 

: tr (z 2fc ) , 1 < k < I , *vz 2fe « , < fc < / - 1 , Pi(z\v); ( ' 

K = 0„,n = K™ ©so„ : 

F,- : tr (z 2fc ) , 2 < 2k < n , l vz 2k v , < 2fc < n - 1 . 



K = Spin 7 ,n = M 8 © R 7 : 

Fl = \v\\F 2 = \z\\ 


(2) 


K = G 2 ,n = R 7 ®R 7 : 

F 1 = \v\ 2 ,F 2 = W,F 3 = |z| 2 . 


(3) 



K = 


U„,n = C n ©A 2 C",n> 3 : 




Fr- 


tr ((zz) fc ) , 2 < 2k < n , v*(zz) k v , < 2fc < n ; 




it = 


SU„,n = C"©A 2 C™,n = 2/,Z>l : 




Fr- 


tr ((zz) fc ) , 1 < k < I , RePf(z), ImPf(z) , v*{zz) k v , < 2k < n ; 




it = 


SU„,n = C"©A 2 C",n = 2i + l,; > 1 : 


(4,5) 


Fr- 


tr ((zz) fe ) , 2 < 2k < n , v*{zz) k v , < k < I , RePf(z|v), ImPf(z|v) ; 


it = 


U 2 ,n = C 2 ©A 2 C 2 : 
F 1 = M 2 ,F 2 = |*| 2 ; 




K = 


SU 2 ,n = C 2 ©A 2 C 2 , : 

Fi = |w| 2 ,F 2 = Rez,F 3 = Imz; 





F = U„,n = C"©u„ : 

Fj : i fc tr (z k ), k = 1, . . . ,n , i k v*z k v, k = 0, . . . ,n - 1 . 


(6) 


F = (Uix)Sp„,n = H"©(F5 ,2 H"©ImH) : 

Fj : tr (z k ) , 2 < k < n , v*z k v, 1 < k < n - 1 , a, 6, c or a, o 2 + c 2 , 

depending on K, where z G ffS'gH and {a, 6, c} is a oasis of Hg. 


(7) 


F = UixSpin 7 ,n = C 8 ©M 7 : 
F x = |z| 2 , F 2 = |«| 2 , F 3 = Re(z(^ 2 )), F 4 = M> 2 | 2 - (ReK« 2 )) 2 , 
where v = v\ + iv 2 accordingly to a Spin 7 -invariant decomposition C 8 = M 8 © M 8 ; 
each R 8 is identified with O and M 7 ImO. 


(8) 


K = S Pl xSp„,n = H"ffispi : 
F 1 = \v\ 2 , F 2 = |z| 2 . 


(9) 


K = Sp 2 xSp n ,n = H 2 <g>EPffisp 2 ,n > 2 : 

Fj : tr(z 2 ), tr(z 4 ), |w| 2 , tr (zv(zv)*), tr {(zvv* - vv*z) 2 ), tr({vv*) 2 ) ; 
K = Sp 2 xSp 1 ,n = H 2 ©sp 2 : 
Fi = tr (z 2 ), F 2 = tr (z 4 ), F 3 = \v\ 2 , F 4 = tr {zv(zv)*), F 5 = tr ((ztw* - vv* z) 2 ) . 


(10) 


F = U 2 xSU„,n = C 2 ®C"ffiu 2 , withn>2 
(in case n>3,K can be replaced by SU 2 xSU„) : 
Fl = itr (z), F 2 = tr (z 2 ), F 3 = \v\ 2 , F 4 = tr (H*) 2 ), F 5 = iv'sv. 


(11) 


F = U 2 xSp„,n = C 2 <X>IHP©u 2 , withn>2: 
Fi - itr (z), F 2 = tr (z 2 ), F 3 = \v\ 2 , F 4 = tr (H*) 2 ), F 5 (x + »y) = |a;| 2 |y| 2 - ( *xy) 2 , 
F 6 = tr (v*izv) , w/iere u = x + iy accordingly to 
an Spj xSp„ - invariant decomposition C 2 ® EP = ]R 4n ffiM 4 ". 


(12) 



Corollary 7.6. If(N,K) is a nilpotent G elf and pair, the K-action on is irreducible, and dim 3 > 1, iften 
the algebra R[n] K of K -invariant polynomials on n is free. 
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Definition 7.7. Suppose that a group G acts linearly on a finite-dimensional vector space V. A subgroup 
H C G is said to be a generic stabiliser of this action if there is an open subset U C V such that the stabiliser 
G v is conjugate to H for all v E U. It is usually denoted by G*(V). 

By a deep result of Richardson, generic stabiliser exists for all linear actions of reductive groups, see 
e.g. [HI Theorem 7.2]. In the following £ stands for a generic point of 3 (or 3(C)), for its stabiliser in A 
(up to a connected component), and A*(n) for the generic stabiliser of the action of A on n. Note that for 
the maximal number of algebraically independent A-invariants, called transcendence degree of E[n]^ and 
denoted by tr.dcglRfn]^, we have the equality tr.degM[n] K = dimn — dim A + dim A*(n). 

Suppose that c C 3 is an analogue of a Cartan subspace, i.e., it is a linear subspace containing a generic 
point £ such that A^ stabilises also all points of c and for a generic point £' g c the intersection A£' n c 
is finite. We do not address here the question of existence of such c. Note only that in the rank one case 
any line R£ with £ € 3 being generic satisfies both properties. Set r := dime. Then r = tr.degM^]^. Let 
F u . . . ,F m be a free generating set in R[n] K with R[F U . . . , F r ] = R[t] K . 

Corollary 7.8. Suppose that (A, A) is a nilpotent Gelfand pair with an irreducible action of A on V, 
dim 3 > 1. and c C 3 is a linear subspace satisfying both assumptions of the previous paragraph. Then 
K[t) © c]^ = M[A r+ i, . . . ,F m ] <E) K[c], where Fj is the restriction of Fj to © c. In particular, the algebra 
R[o © c]^ is free and has the same number of generators, m, as R.[tl] . 

Proof. Consider the composition R[n] A — > R[t> © c] K( — > R.[t> x {£}] x<: of two restriction morphisms. It is 
known to be surjective, see e.g. [3UJ proof of Theorem 1.3]. The polynomials Fx,..,,F r are constant on 
D x {£}. On the other hand tr.degM[D x {C}] Kc = m — r, because a generic A^ orbit on t> has codimension 
to — r. Hence K.[t) x {C}] K<: is generated by the restrictions of Fj with r < j < m and, since the action of Kq 
on c is trivial, R[t) © c] K < = M[A r+1 , . . . , F m ] (8> R[c]. □ 



Let g be a complex simple Lie algebra of rank r and a±, . . . ,a r a set of simple roots with a standard num- 
bering [2H Tables]. We let Wi denote the i'th fundamental highest weight of g and R(wi) the corresponding 
irreducible representation. 

Proof of Theorem \7.5\ We consider each case separately. 

1. The complexification of n is isomorphic to R{vji)®R(w2) as a representation of the complex group SO n . 
This representation can be found in Table 3a, lines 2 and 5, |25j . The description of invariants depends on the 
parity of n. Suppose first that n = 21, i.e., A(C) is of type D and we are looking at line 2 in [251 Table 3a]. 



The set F±, . . . , F m of Theorem 7.5 ^1) has the same degrees (and bi-degrees). Due to Proposition 7.4 it only 
remains to show that the proposed generators are algebraically independent. 

As is well known, the first I polynomials form a generating set in M[so„] so ". In particular, F\,...,Fi are 
algebraically independent. Assume that there is a non-trivial equation Q{F\, . . . ,F m ) = 0. Since the first 
I polynomials are algebraically independent, Q depends non-trivially on the Fj with j > I. Moreover, for 
generic £ 6 3, the restriction to V x {£} gives us a non-trivial equation on the Fj(v) — Fj(v, C) with j > I. 
Suppose w.l.o.g. that ( is a block-diagonal matrix 



diag 



M t 2 \ f ti 

-h '\-h oj'-'U 



with pairwise distinct |tj|. Then F [+k+ i = t\ k (x\ + xl)+tl k (xl + x\) + . . . + t 2 l k {x 2 n _ 1 +x 2 n ), with < k < l-l 
and x\, . . . , x n being coordinates on 0. Because all tf are distinct, the polynomials Fj are linear independent, 
hence, algebraically independent and Q must have been trivial. 

Suppose now that n = 21 + 1. Then A(C) is of type B and we are looking at line 2 in [351 Table 3a]. The 
set F\, . . . , F m of Theorem |7.5[ 1) has the same degrees (and bi-degrees). The proof goes as in the case of 
even n, with the single difference that Fi +k+i is equal to t\ k (x\ +x%) +t\ k (x 2 +x\) + . . . +tf k (x 2 l _ 2 +x 2 l _ 1 ) 
for < k < I — 1 (the sums are linearly independent), and F m = t\t 2 ■ ■ ■ Ux n . Clearly, these / + 1 polynomials 
are algebraically independent. Hence the polynomials Fx, ... , F m are algebraically independent. 



25 



In case K = O n some modification are needed. An additional element acts on most of the SO n -invariants 
trivially. The only exceptions are Pfaffians, which must be replaced by the determinants, or, equivalently, 
by tr (z n ) in case n is even and by t vz n ~ 1 v in case n is odd. 

2. This is a case of the so-called double transitivity, see [17], the group K acts transitively on each product of 
two spheres, one in and one in 3. We can also apply Lemma |7.2| For non-zero a, b 6 C, the level subvariety 
X a ,b = {x £ n{C)\Fi{x) — a,F 2 {x) — b} is a single if(C)-orbit, because the stabiliser K(C) V — G2 of 
a generic v £ C 8 acts transitively on complex spheres in C 7 . It is also quite clear that X a f, ^ for all 
a,b£C 2 . 



3. In this case we use Lemma 7.2 Fix a non-zero value of F3, let us say ^(C), C G 3(C)- The complex group 
Gi acts transitively on the complex sphere Fs(z) — F$[£) in 3(C). Therefore condition (i) of Lemma [7. 2| is 
fulfilled if and only if the stabiliser K(C)^ — SL3 acts transitively on := {v + £ | Fi(v) = a, F2(v + £) = b} 
with a, b £ C. As a representation of SL3 we can decompose o(C) = C C 6 , where C 6 is the orthogonal 
complement to £. Now Xq = {w} x {v £ C 6 | Fi(v ) = a — where *!«C = 6- Such vector w is unique. 

The stabiliser, SL3 acts on Cw trivially, but in the orthogonal complement C 6 = C 3 ©(C 3 )* transitively on 
complex spheres Fi(v) = c, with c 7^ 0. For a generic triple y = (a,b, ^(C)) G C 3 , where a ^ Fi(w), the 
group SL3 acts on transitively, hence, tp~ 1 {y) is a single K (C)-orbits. Clearly <p~ 1 (y) is non-empty for 
all y £ C 3 . Thereby condition (ii) is satisfied as well. 

4,5. At first assume that K = U n and n > 4. Here we use the general theory developed for the multiplicity 
free actions, see beginning of this section. The space n carries a if-invariant complex structure and the action 
of K on C[n] is multiplicity free, i.e., each irreducible representation appears only once C[n] = (J) V\. The 

AeA(n) 

highest weights semigroup A(n) is a free monoid, generated by weights Ai, . . . , A m . The algebra C[n(C)] K ^ 
is freely generated by the invariants sitting in V\ t <8> V£.. Descriptions of A(n) can be found in [16] or [6]. 
In our case m — n and this description leads to the set of generators given in Theorem |7.5[ 4,5). Another 
possible approach could be to take only the degrees of generating invariants from [16] and then prove that 
the Fj's are algebraically independent. 

The case K — SU„ has one additional complication. There are non-constant if-invariant complex poly- 
nomials on the complex space n, e.g., for even n, the determinant of z, for odd n, the determinant of a 
skew-symmetric (n+l)x(n+l) matrix This does not spoil the situation very much, since the algebra 

M[n] x remains free according to [25j Table la, line 11]. Over real numbers the additional invariants are the 
real and imaginary parts of Pf(z) or Pf(z|u), depending on the parity of n. Their product, which is either 
tr ((zz) 1 ) or v*(zz) l v, should be removed from the set of generators. 

Consider now small n. For n = 1, the situation is trivial: n is commutative or a Heisenberg Lie algebra. 
These two cases are omitted from consideration. For n = 2, the centre 3 is C and Rfn]^ is either lR[|u| 2 , \z\ 2 ] 
orM[M 2 ,3]. 

The last case is n = 3. Here the invariants are the same as for n > 4. There is only one simplification, if 
K = SU 3 , n can be identified with C 3 © C 3 and Pf (z\v) is proportional to *vz. Thus the set of generating 
invariants is |w| 2 , \z\ 2 , Im(*uz), Ke( t vz). 

6. The first n invariants form a generating set in M^Un] 11 " or, after complexification, in C[gl r J GL ™. Several 
other facts about these polynomials are well known. For example, for generic s £ C™ the level subvariety 
{£ G g[ ra I Fi(£) — Si,i — l,...,n} is a single GL„-orbit. In order to sho w tha t the proposed invariants 
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Fi, . . . , F m , with m = 2n, generate Rfn]^ we check two conditions of Lemma 

Let t £ Ql n be a regular semisimplc element, i.e., a diagonal matrix with pairwise distinct entries 
(t u . . .,t n ). Set T = (GL„) t . Then T = (C x )" is a maximal torus in GL„. We have o(C) = C" © (C n )*. 
Over C the polynomials v*z k v are proportional to y(z k x), where y £ (C™)*, x £ C™. On V(C) x {t} the 

n 

polynomials F n+ i, . . . , F m reduce to ^ t\xiyi with k = 0, . . . , n— 1, where Xi and yi are suitable coordinates 

i=l 

on C n and (C™)*. Since the t% are pairwise distinct, the Vandermonde determinant is non-zero and the sums 
(restrictions of invariants) span the linear space generated by xiyi. Therefore, the subvariety defined by 
x iUi = Q ^ 0, is a single T-orbit. Hence, for c = {c.\, . . . , c m ) £ C m with c, = Fi{t) for 1 < i < n and generic 



2(1 



with j > n, the level subvariety (p 1 (c) = {v € t)(C) | -Fi(f) = c,}, is a single GL„-orbit. Thus condition 



(i) of Lemma 7.2 is satisfied. 

Each point in C" can be presented as (tr (£), tr (£ 2 ), . . . , tr (£")), where £ £ g[„ is a regular element, i.e., 
such that dim (GL n )c = n. The invariants Fj with i > n on t)(C) x {£} reduce to the pairings (C™)* x C" — » C 
given by the matrices I n , £, £ 2 , . . . ,^ n_1 . Since the elements l n ,£,, ■ ■ are linear independent (that is 

the main property of a regular element), we get n linear independent pairings. Hence for every c £ C™ the 
set {v £ f(C) j Fj(v + £) = Cj~ n ,n + 1 < j < 2n} is non-empty Therefore the set <^ _1 (c) is non-empty for 



all c € C m and condition (ii) of Lemma 7.2 is satisfied as well 



7. Let z be an element in HSqMJ 1 and a, 6, c coordinates on H with a being Ui-invariant. First we look 
on Sp„-invariants in R[H"0ifS' 2 H™]. The complex group Sp 2 „(C), which is the complexification of Sp„, 
acts on n(C) via R(wi) © i?(ra7 2 ). This sum appears as a subrepresentation in [351 Table 4a, item 2] and 
the invariants are described in lines 1,2 of [25J Table 4b]. The representation i?(ra7 2 ) is isomorphic to a go 
action on a subspace fji arising from the symmetric decomposition (Z 2 -grading) g — g © Qi with q = s[ 2n , 
go = sp 2 „(C). The generating Sp 2 „(C)-invariants on gi are known to be the traces of powers, tr(z fc ), with 
k = 2, . . . ,n. On H™ there is only one generator, \v\ 2 . The "mixed" case is dealt with in item 2 of [231 
Table 4b]. The arising invariants have degrees 2 in v and 1, . . . ,n — 1 in z. We have to show that v*z k v is 
not proportional to \v\ 2 tv (z k ) for k = 2, . . . ,n — 1. Indeed if £ G HSqW 1 is a diagonal matrix (t\, . . . , £„) 
with pairwise distinct £i € M and XaLi ti = 0, then v*( k v = Y17=i ^i\ v i\ 2 an< ^ * ne sum ^ s n0 ^ proportional to 

|«| 2 tr(C fc ) = (Eti^)(E i = 1 l« i | 2 )- 

The groups Sp„ and UixSp„ have the same orbits on H™ © /fSgH" and hence the same invariants. In 

case K = Sp n we add to the above polynomials all the coordinates a, b, c, and if K — UixSp„, then a and 

b 2 + c 2 . 

8. Here n(C) decomposes as C 7 © C 8 © C 8 . As a representation of Spin 7 it is R{w\) © 2R(vj^) and is a 
subrepresentation of both items 10 and 11 of [15J Table 3a]. In particular, the algebra of Spin 7 -invariants 
is free. It is not difficult to tell which invariants live on our subrepresentation. There are five generators. 
Identifying C[n(C)] and 5(n(C)), we can say that Fx € S" 2 (C 7 ), F' A , F§ sit in two different copies of S' 2 (C 8 ) 
(one in each), F 2 € C 8 ©C 8 , and F 3 e C 7 © C 8 ©C 8 . The non-trivial action of C* replaces F{ and Fg by 
F i = FlF$. 

In order to express F 3 and F4 over K, we decompose C 8 = M 8 © M 8 as a sum of two real Spin 7 -invariant 
subspaces and identify each of them as the space O of octonions. We also identify 3 = R 7 with ImO. Then we 

write each v e as an octonionic vector f ^ J , with Ui acting by S0 2 -matrices. Now F$(v, z) = Re (z(uit; 2 )) 



2 



and F 4 (u) = \vi\ 2 \v 2 \ 2 - (Re(vi« 2 )) 

9. The group Sp n acts transitively on the spheres in H n = C . Thus Sp„ invariants in this case are 
generated by |i>| 2 and linear functions on ImH. Clearly \v\ 2 is also Spj^-invariant. Hence R[n] K = M[|w| 2 , \z\ 2 }. 
This is another case of the double transitivity [T7j . 

10. First of all, K lf (xC) is trivial in case n = 1 and lf*(tl) = Sp„_ 2 for n > 1; = UixUixSp n . Therefore 
the codimension of a generic F-orbit is equal to 8ri + 10 — (8n + 4) = 6 in case n > 1 and to 5, when n = 1. 
On 3 there are two generating invariants, F\ = tr (z 2 ), F 2 = tr (z 4 ). The representation of F on is a real 
form of gi arising from the symmetric pair (g, g ) = (sp 2 „_|_ 4 ,sp 2 „ffisp 4 ), which is of rank 2, if n > 1, and of 
rank 1, if n = 1. There are two generators: F3 = \v\ 2 and Fq. Over C, the polynomial Fq is the restriction 
to gi of tr (A 4 ), where A is a matrix in sp 2n+4 . Over R, and only in case n > 1, we write Fq as tr ((vv*) 2 ), 
where v is identified with a 2xn quaternionic matrix. 

For the description of the mixed invariants Fn,F§ some other arguments are needed. Note that o(C) = 
C 4 © C 2n and as a representation of the complex group Sp 2n it is a sum of four copies of the defining 
representation C 2 " . According to the First Fundamental Theorem of the so called "classical invariant theory" , 
cf. [221 Subsection 9.3] , the algebra C[o(C)] Sp2 ™ has six generators H±, . . . Hq, all of them of degree 2. These 
generators correspond to non-degenerate pairings between different copies of C 2n . The second (complex) 
group Sp 4 acts on the linear space (Hi, . . . , H e ) c generated by the Hi. In order to understand this action, 
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we compute S 2 (C 4 ® C 2 ") = S 2 C 4 ® S' 2 C 2 " ffi A 2 C 4 ® A 2 C 2 ". The invariants H sit in A 2 C 2 ". Hence the 
action of Sp 4 on (Hi, . . . , H 6 ) c decomposes as a sum C ® C 5 of the trivial and the irreducible 5-dimensional 
representation R(w 2 ). The trivial representation corresponds to the if-invariant F 3 = \v\ 2 . 

Now we are in the situation of item (1) with the action of SO5 on R 5 © SO5. There are six generating 
invariants. In case n > 1 they must be algebraically independent. In case n = 1, the second invariant, of 
degree 2, on ]R 5 is proportional to F 2 . The other five are algebraically independent. The mixed invariants 
are of bi-degrees (1, 2), (2, 2) in H t and z respectively. 

Over R the subspace generated by the Hi can be identified with a hermitian 2x2-matrix vv* . Then the 
Sp 2 -invariant norm is defined as |wu*| 2 = tr ((vv*) 2 ). The action of sp 2 is z-vv* — zvv* — vv*z. Therefore, 
the invariant of be-degree (2, 2), which is \z-h\ 2 (h e (Hi, . . . , H 6 ) R ), should be written as tr ((zvv* — vv*z) 2 ). 
It only remains to show that F4 = tr (zv(zv)*) is not proportional to FiF 3 . 

Chose a generic ( G 3. We may assume that it is given by a pair (2*1,^*2) of quaternions with *i,*2 G M 
and |ti I 7^ |*2 1- Under the action of the stabiliser = Sp 1 xSp 1 xSp„ the subspace decomposes as 
= H n © H n . Let us split v accordingly to this decomposition v — Vi + v 2 . Here both vi and v 2 are rows. 
Then the restrictions of Fi ,F 3 , and F4 to x {£} are proportional to 

tj + t 2 2 , M 2 + H 2 , 4\vi\ 2 + t 2 2 \v 2 \ 2 . 

Since |ti I and |* 2 | are distinct, * 2 |^i| 2 + tl\v 2 \ 2 is not proportional to (t\ + *|)(|i>i| 2 + | ^2 1 2 ) - 

11. Here Kq = (Uix)UixSU„, K*(n) = (Uix)SU„_ 2 for n > 2 and K*(n) is trivial for n = 2. Note that 
the (additional) torus lies in a generic stabiliser, hence, the invariants do not differ. Thus we may (and will) 
assume that K is not scmisimplc. (In case n = 2 the torus is necessary.) 

The codimension of a generic orbit is 5. On 3 there are two generating X-invariants: F\ = itr (z) and 
F 2 = tr(z 2 ). The subspace is a real form of Qi arising from a symmetric decomposition fl = go © 01 
with (g,0o) = (si n +2,sl n (Bsl 2 ). Here there are two generators F3 = \v\ 2 and det(iw') or, equivalently, F4 = 
tr((vv*) 2 ). Thus R[s] K = R[F 1 ,F 2 ] and R[v] K = R[F 3 ,F 4 }. In order to describe the "mixed" generators, we 
explore the same method as in item 10. 

We have t>(C) = C 2 <x€™ © (C 2 ®C")* and C[o(C)] GL " = C[H U H 2 , H 3 , H A ] with all four generators being 
of degree 2. On the subspace (Hi, . . . , H A ) C the group SL 2 acts as on C ffisl 2 . The trivial subrepresentation 
corresponds to F 3 . On the other hand, Fi comes from the trivial subrepresentation of K in 3. Apart from 
these two generators, we have an action of SL 2 on sl 2 © sl 2 or of SO3 on C 3 © C 3 . This gives rise to three 
invariants: one on the first copy of C 3 , another on the second copy, and one "mixed", Fg, of bi-degree (2, 1) 
in (0,3). Thereby R[n] K is generated by R[V] K , R[i] K , and F^. The subspace (S 2 (V(C)) <g> }(C)) K( -V is 
two-dimensional and has a basis {Fg, ^1^3}. In order to complete the description, it is sufficient to show 
that FiF 3 and F$ are linear independent. Indeed, if tr (z) = 0, then FiF 3 = 0, but F5 is not. 

12. Here = UixUixSp n , K t (n) = Sp„_ 2 , and the codimension of a generic orbit is 6. This case is 
very similar to case 10. The complcxification o(C) = (C 2 ©(C 2 )*) <X> C 2 " is the same as for in item 10 and 
the -KT(C)-representation on t>(C) is obtained by the restriction of the defining representation of Sp 4 (C) to 
a symmetric subgroup GL 2 . Again C[o(C)] Sp2 " = C[Hi, . . . ,Hq], where all generators are of degree 2, and 
GL 2 acts on (Hi, . . .,H 6 ) C as on A 2 (C 2 ffi(C 2 )*), i.e., as on C ffi (C®C*) ffis[ 2 . On the first summand C the 
action is trivial and it corresponds to F3 = \v\ 2 . On 3 there are two generators: Fi — tr (z),F 2 = tr(z 2 ). 
Apart from this there are two invariants of degree 4 on 0, one comes from C C* , another from s[ 2 ; plus an 
invariant Fq of bi-degree (2, 1) in (0,3), which corresponds to the pairing between sl 2 C (Hi, . . . , H 6 ) c and 
sl 2 C 3(C). There are at most 6 generators. Since we should have at least 6 algebraically independent ones, 
they are algebraically independent. As F 6 we take (zv,v), where ( , ) is a F-invariant scalar product on 0. 
Clearly F e is non-zero on x sl 2 and hence linear independent with FiF 3 . The polynomial can be expressed 
as F 6 = tr (v*izv). 

To complete the proof we need to show that R[v] K = R[F 3 , F 4 , F 5 ]. Recall that F 4 = tr ((vv*) 2 ). This is 
an invariant of K and, taking into account an isomorphism C 2 <X>c H n = H 2 (gin H", one concludes that it is 
also Sp 2 xSp„-invariant. 
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On the other hand, C 2 <Ei HP = C <8>r HP as a representation of Ui x (Spj^xSp^) = K. One may say 
that is a sum of HP = M 4 ™ and iR 4n , where Ui acts via multiplications by cos 7 + isin7. Such situation 
appeared in case 8. This decomposition leads to the construction of F 5 . Since the degrees of generating 
invariants have already been found out, it remains to show that ^3,^4,^5 are algebraically independent or 
that H 4 ,-F 4 ,F 5 are linearly independent. If they were dependent, then the complex groups Sp 4 x Sp 2n and 
C x S04„ would have had the same invariants on C 4 ® C 2 ™ = (C®C*) <g) C 4 ™. But a subalgebra of S08n(C), 
generated by the Lie algebras of these two subgroups, coincides with S08n(C) itself and neither F4 nor F5 is 
S08n(C)-invariant. Therefore F3, F4, and F5 are algebraically independent and this completes the proof. □ 

Remark 3. For the purposes of Section 4 we need a slightly modified set of generators in case (12). As was 
already mentioned, t> can be viewed as K 2 (8>g HP and K as S02xSp 1 xSp n . Also 3 decomposes as sp 1 © K. 
Now F\ = t, where t is a coordinate on E C 3 and the new F 2 — \z \ 2 with z £ sp 1 . The next generator, 
F3, we leave as it is. We can present set) asa2xn quaternionic matrix, 



c = 



^1 
v 2 



where vi,V2 G H™ are row- vectors. It can be assumed that l v 2 are the same as x,y in Theorem 7.5 
In this terms F4 = |«i| 4 + 2|t; 1 t!2| 2 + 1 4 - There is a unique (up to a scalar) Sp 1 x Sp n -invariant pairing 
between HP and HP. In terms of real column- vectors x, y it was expressed as l xy. In terms of quaternionic 
row vectors v\ , v 2 it becomes exactly Re (vi v 2 ) . Hence F 5 = \ vi \ 2 \ v 2 \ 2 — (Re (^1^2)) ■ For the new generator 
Fq we take a /^-invariant polynomial Ke((viv' 2 — v 2 v'l)zo) (here again zq g spj). As well as the old Fq of 
Theorem 7.5 Re {(viv 2 — v 2 vl)z ) is non-zero on x sp x and hence linearly independent with FiF 3 . 
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